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Words of Wisdom 


June 9, 2015 
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"If can only be attributable to human error/' 

- HAL 9000 (2001: A Space Odyssey) 



NASA Lessons Learned 



• NASA's Recent Losses in Space and on the Ground 

- Failure is not an option we want to choose, but it is a reality.... 

• The NASA Safety Culture 

- Using the Safety Culture Model to Analyze NASA's History 

- Measuring Safety Culture 

- Safety "Beyond the Numbers" 

• A Rejuvenated Risk Management Environment 

- Risk informed decision-making 

- A process for identifying and addressing dissent 

- Improved risk management processes in mission planning.... 

.... and out in the trenches 
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NASA's Losses 


Recent Mission Mishaps 


NOAA N-Prime, 
September 6, 
2003: 

• $135 Million 
vehicle damage; 

• 5.5 year mission 
impact. 




Columbia STS-107, February 1, 2003: 

• 7 fatalities; 

• $3 Billion vehicle loss; 

• 2.5 year mission impact. 


OCO, February 24, 2009: 

• $280 Million vehicle loss; 

• 5+ year mission impact. 


Glory, March 4, 
2011: 

• $424 Million 
vehicle loss; 

• ??? mission 
impact. 


Genesis, September 8, 2004: 

• Some sample retrieval materials lost. 
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DART, April 16, 2005: 

• Proximity operations 
mission objectives 
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NASA's Losses 


Recent Institutional Mishaps 



KSC Roofing Fatality, 
March 17, 2006 

• Subcontractor died 
from head injuries 
suffered due to fall. 


JSC Chamber B 
Asphyxiation, 
July 28, 2010 

• Shoulder 
injury due to 
asphyxiation 
and fall. 



June 9, 2015 




MSFC Freedom Star Tow-wire Injury, December 12, 2006 

• Hospitalization due to internal injuries from impact with SRB 
tow-wire. 



WFF CNC Injury, 
October 28, 2010 

• Sub-dermal 
tissue damage 
due to impact 
from machine 
tool shrapnel. 




What is the impact of Human Factors 

and Safety Culture on the Mishap Environment? 





* Estimates range from 65-90% of catastrophic mishaps are due 
to human error. 

- NASA's human factors-related mishaps causes are estimated at ~75% 


* As much as we'd like to error-proof our work environment, 
even the most automated and complex technical endeavors 
require human interaction...and are vulnerable to human 
frailty. 


* Industry and government are focusing not only on human 
factors integration into hazardous work environments, but 
also looking for practical approaches to cultivating a strong 
Safety Culture that diminishes risk. 

June 9, 2015 David T. Loyd 7 



TRUST LEVEL and CLARITY 


Addressing Barriers to Trust 


nV^A 



Positive Safety Culture begins 
with assuring dialog is open and 
decision-making is transparent. 
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ISSUE RESOLUTION FORUMS 




The NASA Safety Culture 




"/ believe that this nation should commit itself to 
achieving the goal, before this decade is out, of 
landing a man on the moon and returning him safely 
to the earth." 


• Space-related tragedies have marked 
our safety culture evolution. 

• It's not possible to perpetuate a safety 
culture in space without taking care of 
each other on the ground and at home. 
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The NASA Safety Culture 



NASA Safety Culture Working Group, consisting of 
membership from each NASA Center, has been active 
since early 2009. 

* NASA's Definition of Safety Culture - 

“An environment characterized by safe attitudes and 
behaviors modeled by leaders and embraced by all that 
fosters an atmosphere of open communication, mutual 
trust, shared safety values and lessons, and confidence 
that we will balance challenges and risks consistent with 
our core value of safety to successfully accomplish our 
mission. ” 
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NASA's Safety Culture Model 


An effective safety culture is characterized by 
the following subcomponents: 

Reporting Culture 

We report our concerns 

Just Culture 

We have a sense of fairness 
Flexible Culture 

We change to meet new demands 
Learning Culture 

We learn from our successes and mistakes 

Engaged Culture 

Everyone does his or her part 
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Catastrophic Event Impact 

Using the Safety Culture Model to Analyze NASA's History 


Apollo 1 - January 27, 1967 

Reporting - Procedures were subjected to last- 
minute changes that were not tracked, recorded 
or communicated. 

Just - Absence of information on this factor attests 
to the general neglect at the time of 
organizational behavior as a key factor in 
mishaps. 

Flexible - Willingness to change was weak in the 
presence of compelling important information. 

Learning - NASA failed to appreciate the significant 
hazards of a 100% oxygen environment. 

Engaged - NASA provided insufficient surveillance 
over its own management functions. 
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Catastrophic Event Impact 

Using the Safety Culture Model to Analyze NASA's History 



Apollo 13 -April 13, 1970 

Reporting - Incomplete and sometimes incorrect 
information was used in problem solving. 

Just - Absence of information on this factor attests 
to the general neglect at the time of organizational 
behavior as a key factor in mishaps. 

Flexible - Demonstrated ability to adapt quickly to 
an emergency although flexibility prior to the 
mishap is unclear. 

Learning - While safeguards had been implemented 
following the Apollo 1 fire, key aspects of design, 
workmanship, and material use remained 
vulnerable to oxygen flammability. 

Engaged - Solutions immediately following the 
oxygen tank explosion represent an engaged team. 
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Catastrophic Event Impact 

Using the Safety Culture Model to Analyze NASA's History * * : *• 



June 9, 2015 


David T. Loyd 


Challenger - January 28. 1986 

Reporting - Ineffective problem reporting 
requirements and practices. 

Just - Stifled communication regarding O-ring 
susceptibility to cold conditions. 

Flexible - Launch concerns were dismissed in 
the face of significant schedule pressure. 

Learning - Trend analysis was inadequate as 
evidenced by identification of a number of 
burn-through events which occurred prior 
to STS-51L. 

Engaged - NASA management lacked 
involvement in critical discussions. 



Catastrophic Event Impact 

Using the Safety Culture Model to Analyze NASA's Histor 
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Columbia - February 1. 2003 

Reporting - Foam shedding was a known problem, yet foam impact data was sill 
being analyzed at the time of the flight, and not considered a serious hazard. 

Just - Some engineers were reluctant to raise concerns when faced with a return of 
an "in God we trust - all others bring data" attitude. 

Flexible - Like the Challenger mishap, the Shuttle Program was experiencing 
schedule pressure challenges. 

Learning - With "normalization of deviance," foam had become classified as "in- 
family" and as a negligible risk to the orbiter. 

Engaged - "Echos" of the Challenger mishap were evident. 
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Measuring Safety Culture 


Safety Culture Element Results 


Very 
Satisfied m 


Satisfied {5) 

Slightly 
Satisfied <4} 


Slightly 
Dissatisfied m 


Dissatisfied {2J 
Very 

Dissatisfied {1J 


-rU 
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Learning Engaged 


10 11 12 13 14 15 


#13 Most Significant Safety Culture 
Element 


43 % 



24 % 


■ Report ■ Just Flexible Learning Engaged 


JSC Results: 

• Ratings were VERY similar with KSC and MSFC - ratings in the satisfied range. 

• "Reporting Culture" was rated highest, while "Engaged Culture" was valued most. 

• 97% of JSC employees feel safe working at the Center. 

• 95% of JSC employees agreed they are responsible for their safety and their co-worker's safety 

• Comments were both positive and negative. 

• Results have been communicated with senior staff and multiple employee forums. 
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Safety "Beyond the Numbers" 


Leadership 

Continue to encourage safe behaviors, attitudes, 
and employee involvement. 


Prevention 

Improve employee participation in prevention 


activities. 


Percent Inspected by Directorate - FY10 





JSC Safety & Health - Close C 



1 Qtr 2 Qtr 3 Qtr 4 Qtr 1 Qtr 2 Qtr 3 Qtr 4 Qtr 1 Qtr 2 Qtr 3 Qtr 4 Qtr 1 Qtr 2 Qtr 3 Qtr 4 Qtr 1 Qtr 2 Qtr 
2006 2006 2006 2006 2007 2007 2007 2007 2008 2008 2008 2008 2009 2009 2009 2009 2010 2010 


Reaction 

Reduce mishaps and improve investigation 



Issue Resolution 

Assure response to challenges reflect 
thoughtful approach to risk mitigation. 


GSE 


PV/S 


Other 


Facility 


Motor 

Vehicle 


PA-1 -- Challenging, safe and successful collaboration 
between: 

• Johnson Space Center 

• Dryden Flight Research Center 

• Langley Research Center 

• JSC White Sands Test Facility 

• US Army White Sands Missile Range 

• Orbital Sciences 

• AlliantTechsystems 

• Aerojet 
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NASA Decision Model 



ALTERNATE I 


isserrtint 



IMPLEMENT DECISION 

■ Document decision. (IPK) 
Notify affected parties 

■ Follow Program or 
Organization-specific 
implementation process 
Track Action Items 


DECISION 

MAKER 



BRIEF DECISION MAKER 
identify who needs to be involved in decision 
Identify Decision Purpose 
Identify Time Cnticaiity 
Present and Discuss Issue 
* Presentation Templates 
Poll BoardfPanel/Working Group 


END 

Verify desired 
outcome 
is achieved 


DEFINE THE ISSUE: PROBLEM, 
DECISION, OR OPPORTUNITY? 
Define What, Where, When, 
Extent & Impacts 
Identify the Causes 
Perform inspection, Test, or 
Analysis 

Identify Possible Solutions & 
Perform Risk Assessment 
Identify Time Criticality 



PREPARE DECISION MAKER 

* Data Review 

* Write a White Paper (DOC) 

* Pre-Board Meeting 


DOCUMENT 
ANALYSIS TO FILE 


\a. 
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NASA Dissenting Opinion Process 


NASA Policy Directive 1000.0, " NASA Governance and 
Strategic Management Handbook" 

• In assessing a decision or action, there are three choices: 

1. agree, 

2. disagree but willing to support, 

3. or disagree and raise a Dissenting Opinion. 

• A "Dissenting Opinion" is a substantive disagreement 
with a decision that is judged not in the best interest of 
NASA. 

• A Dissenting Opinion must be supportable and based on 
a sound rationale. 

June 9, 2015 David T. Loyd 19 
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NASA Dissenting Opinion Process - Resolution 

* ’ * 


Key steps of the Dissenting Opinion resolution process 

are: 

1. Disagreeing parties must jointly establish the facts 
agreed upon; 

2. The parties jointly present to the next higher level of 
authority; and 

3. If the dissenter is not satisfied with the process or 
outcome, the dissenter may appeal to the next higher 
level of management. The dissenter has the right to 
take the issue upward through the organization, even 
to the NASA Administrator, if necessary. 

June 9, 2015 David T. Loyd 20 



Risk Assessment Concepts & Requirements 





NPR 8000.4, Agency Risk Management Procedural Requirements 


Risk Informed Decision-Making (RIDM) 
involves: 

(1) Identification of decision alternatives , 
recognizing opportunities where they 
arise , and considering a sufficient number 
and diversity of performance measures to 
constitute a comprehensive set for 
decision-making purposes. 

(2) Risk analysis of decision alternative, 
support ranking. 

(3) Selection of a decision alternative 
informed by (not solely based on) risk 
analysis results. 


Risk Management 



RIDM within an Organizational Unit 


Identification of Alternatives 


i 


Risk Analysis of Alternatives 

♦ 


Risk-informing Alternative Se/ccrton 


T 


Baseline Performance Requirements 


Elevate Decision One Level Up it Needed 
and 

Report Top Risks one Level up 


^RM within the Organizational Unit Tasked^ 

Implement the Selected Alternative 



Identify 
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CRM Feedback to RIDM 





Probabilistic Risk Assessment (PRA) 





• PRA integrates models based on systems 
engineering, probability and statistics, 
reliability and maintainability 
engineering, physical and biological 
sciences, decision theory, and expert 
opinion. 

• PRA is needed when decisions need to 
be made that involve high stakes in a 
complex situation. 

• The collection of risk scenarios allows 
the dominant risk factors to be 
identified, then modified or eliminated 
to improve the probability of success. 



Representing the World via Bayesian Inference. 


May 14, 2014 
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Person-Risk Per Year 


Truck Driver 


Timber Cutting and Logging 


Airline Pilot 


1:3790 


1:998 


1:1270 


Alaskan Commuter Pilot 


1:336 


Construction Worker 


1:4190 


Miner (Not Oil and Gas) 1:2500 

1:851 


Commercial Fishing 

Alaskan Commercial Fishing 

Northeast Multispecies 
Groundfish Fishing 

Shuttle Astronaut 


1:775 


1:166 


Miner risk does not include fatalities due to 
chronic illnesses like "black lung" 

Risk increases as "drill down" into 
smaller and smaller groups that drive 
the risk. Shuttle Astronaut risk is a very 
small group that has high risk 


1:218 


1:100 


1:50 


1:33 


Probability 


Simplified Risk Management Flow 


Identify 


Assess risk 

^<J b S i NO ^^iTthe co^t^^YES 

Accept 

risk 

— > 

probability 

— XC J f* t0 >-K of risk >4 

regulatory / ' \ tolerable? ^ 

and 



and severity 

^\control?x^ ^^toieraoie^^ 

monitor 

k A 






^ Avoid risk ^ 


NO 


Is risk 
elimination 
cost- 

effective^ 
NO 

Can risk 
severity be 
reduced? 

NO 

Can risk be 
delayed? 

NO 


Can risk be 
transferred? 


YES 


YES 


YES 


YES 


Control 

and 

monitor 


Mitigate 

and 

monitor 


Delay and 
monitor 


Transfer 

and 

monitor 
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Risk Scorecard 



LIKELIHOOD RATING 

L 

K 

E 

L 

H 

0 

0 

D 

5 

Very 

| 

Expected to happen. Controls have minimal to no effect. 

4 

Likely 

Likely to happen. Controls have significant limitations or 
uncertainties. 

3 

Possible 

Could happen. Controls exist, with some limitations or 
uncertainties. 

2 

Unlikely 

Not expected to happen. Controls have minor limitations 
or uncertainties. 

1 

Highly 

Unlikely 

Extremely remote possibility that it will happen. Strong 
controls in place. 





SEVERITY 


High — Mtigate; implement new 
processes, change requirements, 
or re-baseline 


□ 

□ 


Moderate — Manage/consider 
alternative processes, or Accept 

Low — Manage within normal 
processes; or Close 


CONSEQUENCE 

Sub categories 

1 

2 

3 

4 

5 


Personnel 

Minor injury; 
Minor OSHA violation 

Short-term injury; Moderate 
OSHA violation 

Long-term injury, impairment 
or incapacitation; 
Significant OSHA violation 

Permanent injury or 
incapacitation; 
Major OSHA violation 

Loss of life 

HSE 

(Health, Safety, 
Environment) 

System, Facility 

Minor damage to asset 

Moderate impact or 
degraded performance 

Loss of non-critical asset 

Damage to a critical asset 

Loss of critical asset or 
emergency evacuation 


Environment 

Minor or non-reportable 
hazard or incident 

Moderate hazard or 
reportable violation 

Significant violation; Event 
requires immediate 
remediation 

Major violation; Event causes 
temporary w ork stoppage 

Catastrophic hazard 

TECHNICAL 

Performance 

Minor impact to mission 
objectives or 
requirements 

Incomplete compliance w ith 
a key mission objective 

Noncompliance; Significant 
impact to mission 

Noncompliance; Major impact 
on Center or Spaceflight 
mission 

Failure to meet mission 
objectives 

CENTER 

CAPABILITIES 

Infrastructure 

Minor impact or reduced 
effectiveness 

Moderate impact or damage 
to infrastructure 

Significant damage to 
infrastructure or reduced 
support 

Mission delays or major 
impacts to Center operations 

Extended loss of critical 
capabilities 

Workforce 

Minor impact to human 
capital 

Moderate impact to human 
capital 

Significant impact; Loss of 
critical skill 

Major impact; Loss of skill set 

Loss of Core Competency 

COST 

Organizational or 
CMO Impact 

<2% Budget increase or 
<$1M CMO Threat 

2-5% Budget increase or 
$1M-$5M CMO Threat 

5-10% Budget increase or 
S5M-10M CMO Threat 

10-15% Budget increase or 
$10M-$60M CMO Threat 

>15% Budget increase or 
>$60M CMO Threat; 

SCHEDULE 

- 

Minor milestone slip 

Moderate milestone slip; 
Schedule margin available 

Project milestone slip; No 
impact to a critical path 

Major milestone slip; Impact to 
a critical path 

Failure to meet critical 
milestones 





HSE Panel Review (Risk Validation) 

* * * 


Objectives : 

• Better understanding of institutional risks with health, safety, environment (HSE) consequences 

• Center-level consistency in HSE consequence assessment 

• Better inform Center risk decision process about severity of HSE consequences 

Approach : 

• SMEs review risks in the HSE domain, and associated mitigation parameters 

• SMEs advise risk owners and the JSC Risk Management Working Group (RMWG) 

• Direct involvement by SMEs from Occupational Safety & Health, Environments, Facility Safety, 
and other disciplines in determining HSE consequence severity, analysis, and mitigation 

Benefits 

• Helps risk owners assess consequence severity in HSE domain and improve mitigation plans 

• Introduces quantitative methods commensurate to risk / uncertainty levels 

• Helps risk-inform Center-level decisions related to: budget allocations, unfunded mandates, and 
compliance/ noncompliance with regulations, etc. 

• Helps aggregate risks and consequences in the HSE domain 
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Institutional Risk Management 





L 5 
I 

K 4 
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H 
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O 
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1 2 3 4 5 

CONSEQUENCE 


Legend 

▲ Top Center Risk (TCR) 

A Proposed Top Center Risk (Proposed TCR) 


LxC 


3x4 


4x5 


4x3 


4x3 


5x5 


4x4 


3x4 


4x4 


4x4 


5x4 


Title 

(Notional RiskTitiles) 


Test system maintenance 


Mission essential resource limitations 


Equipment End-of-Life 


Building Refurbishment: 


Comm Systems 


Buildi 


Pater System-Repairs/Upgrades 


A Research equipment failure threat 


Consequence 



## 
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Risk/Hazard Identification Processes 


Pre-Operational Phase 

* Hardware & mission planning 

* Probabilistic Risk Assessment 

* Facility development 


Design/ 

System 

Hazards 



Behavior/ 

Performance 

Hazards 


Environment/ 

Condition 

Hazards 


Operational Phase 

* Job Hazard Analysis 

* Workplace Inspection 

* Facility Risk Assessment 

Process Hazard Analysis 
System Safety Approach 

Survey/Inspection 


Post-Accident Phase 

• Mishap Investigation 

• Corrective Action Plan 


Close Call Investigation] 


Job Hazard Analysis 

S 







W: 


[Mishap 

[investigation 


03/2/05 


Technical Scope 
Medium Mesh 


Broad Scope 
Course Mesh 


Focused Scope 
Fine Mesh 


Wide Scope 
Course Mesh 



Targeted Scope 
Variable Mesh 



Process Measures for High-Risk Facilities 


• Worldwide industry and government organizations have developed 
effective indicator programs, recognizing the value of leading indicators at 
reducing the risk of catastrophic mishaps. 

• In the US, events such as the BP Texas City explosion and the Deep 
Water Horizon spill have compelled action to develop a standard for 
process safety-related leading indicators. 

• Examples of leading measure areas for high-risk systems include: 

- Maintenance and system integrity conditions 

- Operational qualifications 

- Challenges to safety systems and monitoring equipment 

- Communication and reporting system conditions 

- Accuracy of configuration management 

- Maintenance of operational procedures and emergency response plans 

• NASA has adapted this approach to assess risk controls associated with 
hazardous, critical, and complex infrastructure. 
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"...Required life cycle safety program tasks for facilities judged as hazardous, critical or complex as a result of risk 
assessment." 


High-Risk Facilities 



Process requirements for hazardous, 
critical, or complex facilities: 

• Organizational responsibilities, 

• Personnel training, 

• Operating procedures, 

• Configuration management, 

• Maintenance, 

• Resources, schedules and milestones, 

• Integration with other program 
engineering and management activities. 
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Facility Risk Criteria 

Hazardous: 

Facilities, by their standard 
operation/mission, subject 
personnel to risks/hazards that are 
not normally seen in the standard 
workplace environment. 

Critical: 

• Unique, irreplaceable facilities 
that support space flight 
activities. 

• Facilities supporting unique 
facilities that provide utility 
services. 

• Facilities which contain 
historically significant national 
treasures. 

Complex : 

• Require multiple organizations 
to conduct facility mission. 

• Require extensive employee 
training. 

• Have integrated systems using 
specialty and prototype 
equipment. 

• Contain equipment that is 

specifically designed and high 
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Low-Risk Facilities 


Places of 
assembly 


Office 

occupancies 


Facilities which do not meet the hazardous, 
critical, or complex criteria are subject to 
compliance with regulatory standards and 
national consensus codes. 
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Facility Safety Risk Control Assessment , 



J 



Building/Facility identifications 
L_ / 


Ellington 

Field 
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Mission 

Operations 


— I — 

Engineering 




IRD COD 


SA KA 


NS 


OA 


Characteristic 
Assessment Key 


Elements of 
Chapter 10.4 are 
not applicable to 

Applicable the associated 

facility mission. 

2013 HATS 

Items identified as 
nonconforming in 
2013 were 
resolved. 

Closed: 

Conforms 

* Non- 
conformance 

Documentation 
does not exist to 
support the 
requirements of 
Chapter 10.4. 

F’artially 

conforms 

Significant 
information is 
available, but 
does not meet the 
intent of Chapter 
10.4, or it is out of 
date or 
unavailable. 

Conforms 

Documentation is 
available with the 
required 
information to 
meet Chapter 
10.4. 
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* In most instances non- 
conformances represented 
potentially uncontrolled RAC 3 
or 4 hazards. 






Industry Hazard Index 


Facility Risk Benchmarking with Insurance Industry 





Overall Risk Score Considers: <u 7 

o 60 

• Industry Hazard Index o 50 

• Weighted at 10% ^ 40 

• Hazard Deviation (JSC Mishap Rates) J2 30 

• Weighted at 30% ^ 20 

• FBD Score (Risk Control) 

• Weighted at 60% 



Industry Hazard Index: 

Industry Index is determined by the type of 
operation (by NAICS code) for each high- 
risk JSC facility. 

Incident rates are scaled 0-100 considering 
max value across all industries (not just 
those associated with JSC facilities). 

For example, if the industry hazard index is 
30, then 30% of industries are less risky. 


Comparative Risk of JSC Facilities 



1 1 1 1 1 


El 35 E990 E276 E271 E245 920N 222 319 35 221 
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Rules~of Thumb for Managing Risk ^ 

. * 


• Accept risk only if it is low enough to tolerate 
and within regulations. 

• NO ONE GETS HURT!!! 

• Tolerate only the damage you can afford. 

• Avoid risks you don't NEED to take. 

• Risks change as often as the facility, people, and 
processes associated with them, so they must be 
monitored and reassessed periodically. 
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Backup Charts 
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Columbia STS-107, February 1, 2003: 

• 7 fatalities; 

• $3 Billion vehicle loss; 

• 2.5 year mission impact. 


Kalpana Chawla 
Rick D. Husband 
Laurel B. Clark 
Nan Ramon 
Michael P. Anderson 
David M. Brown 
William C. McCool 
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NOAA N-Prime, September 6, 2003: 

• $135 Million vehicle damage; 

2015 • 5.5 year mission impact. 
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Orbiting Carbon Observatory, 
February 24, 2009: 

• $280 Million vehicle loss; 

• 5+ year mission impact. 
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Glory, March 4, 2011: 

• $424 Million vehicle loss; 

• ??? mission impact. 
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JSC Chamber B Asphyxiation, 
July 28, 2010 

• Shoulder injury due to 
asphyxiation and fall. 



Ion Gauge 
Location approx 
10' above floor 


iployeestood here 
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